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ABSTRACT 



The Job Training Partnership Act's Out-of -School Youth 
program (OSY) was audited to determine whether participants' reported 
outcomes for positive terminations were accurate and fully documented and 
what impact program interventions had on participants' post-program earnings. 
Program services and outcomes were audited for 499 OSY participants at 34 
randomly selected Service Delivery Areas. Findings indicated that positive 
program outcomes that were sufficiently documented were significantly lower 
than those reported; a significant number of entered unsubsidized employment 
terminations were reported in error and others were in question; 
documentation supporting youth employability enhancements was seriously 
deficient; participants' post-program earnings were affected by program 
interventions and level of participants' participation; post-program earnings 
were directly impacted by the type of training a participant received; and 
almost half of the training activities to address participants' barriers were 
not completed. (Appendixes include primary criteria for youth employability 
enhancements; sampling plan for the audit; and Deputy Assistant Secretary's 
response to the draft report.) ( YLB) 
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EXECUTIVE SUMMARY 



Because one of the Secretary’s top Fiscal Year (FY) 1999 priorities was youth services, 
the Office of Inspector General (OIG) audited the Job Training Partnership Act’s 
(JTPA) Title 

II-C out-of-school youth (OSY) program's performance for Program Year (PY) 1997, 
July 1, 1997, through June 30, 1998, the last PY of outcomes information available at the 
time we began our audit. 

Our audit objectives were to determine: 

• whether out-of-school youth participants’ reported outcomes for 
positive terminations were accurate and fully documented (see 
finding 1); and 

• what impact program interventions had on participants’ post- 
program earnings (see finding 2) . 

With respect to our first objective, we determined that positive reported outcomes for 46 
percent of the participant sample (228 of 499) we evaluated were not documented. This 
determination is not intended to imply that only documented outcomes are valid positive 
outcomes. Some of the outcomes that were not documented may have, in fact, been 
attained. However, unless a reported outcome is documented, the outcome’s veracity is 
not certain. 

In our opinion, based on the significance of the rates of positive outcomes that were not 
documented, the OSY outcomes rates reported by the Standardized Program 
Information Report (SPIR) and included in the PY 1997 Chief Financial Officer (CFO) 
Report are significantly lower than reported. The JTPA Title II-C OSY program’s true 
effectiveness is probably somewhere between the Employment and Training 
Administration’s (ETA) reported rates and the OIG’s documented successes; i.e., 
positive outcomes are lower than ETA reported, and some of the outcomes the OIG 
found not to be documented were no doubt attained. However, unless the outcomes are 
documented, the true effectiveness of youth training programs cannot be determined. 

Performance information cannot be relied upon for decision making and other purposes 
unless it is known to be accurate. The performance information included in the 
Department’s CFO Report was prepared pursuant to the Government Performance and 
Results Act (GPRA) , which was enacted in part to improve congressional decision 
making by having agencies objectively inform Congress of Federal programs’ results. 
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To ensure reported data are reliable, GPRA requires that agencies establish means to 
validate measured values that are presented in the annual performance report. ETA 
recognizes the importance of being able to substantiate its externally reported 
information and has initiated a data validation study of information reported under the 
Workforce Investment Act (WIA) , which replaced JTPA. Had such measures been in 
place for the Title II-C performance information we audited, our results probably would 
have been different. However, neither ETA nor the grantees consistently applied 
procedures to assure the validity of the data we audited, and we found significant 
discrepancies between what was reported and what we could substantiate. 

Actions must be taken by all parties - the service deliverers, Workforce Investment 
Boards, states, and the ETA - involved in the current WIA program to verify youth 
performance outcomes to ensure they are accurate and fully consistent with WIA 
regulations and GPRA requirements. 

Related to our second objective, youth who participated in occupational skills training 
(OST) activities had earnings in the 2 years following termination that were twice the 
level of those who did not participate in OST. In addition, for those OST participants 
who completed the activity, post-program earnings for the 2-year period after termination 
were 69 percent higher than for those who did not complete the OST activity. 

Also related to our second objective, we tried to determine whether: (1) the Individual 
Service Strategy (ISS) was being used to provide training or services to specifically 
address participants’ employment barriers; and (2) the provided interventions impacted 
post-program earnings. While the majority of participants were enrolled in activities to 
address their barrier to employment, the participants did not complete approximately half 
the activities. 

We recommend the Assistant Secretary for Employment and Training: 

• Notify all states and substate WIA grantees of the audit results and 
emphasize the necessity for states and their grantees to validate 
outcomes data as part of their monitoring programs and training 
provider eligibility determinations. 

• Emphasize to the states and substate grantees the importance of 
documenting not only reported outcomes but also specific services 
provided, dates services were provided, and actual program exit 
date - date of last activity. 

• Include in ETA’s monitoring program a review of the states’ and 
substate WIA grantees’ implementing such a data validation effort. 
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• Emphasize the importance of the WIA program’s not only enrolling 
youth in occupational skills training activities, but also finding ways 
to keep them actively participating in the program to completion, to 
maximize their post-program earnings. 

ETA’s Response to Draft Report and Auditor’s Conclusion 

The Deputy Assistant Secretary agrees that performance results must be credible. The 
Deputy Assistant Secretary indicated that if the reality or the perception is that the programs 
are not reporting accurately, the system s integrity is at stake. Therefore, the Employment and 
Training Administration (ETA) is committed to developing clear, precise, and feasible 
standards for documenting program outcomes. ETA s new data validation initiative is 
attempting to create both more precise programming specifications and also more precise 
standards for validating data quality. 

The OIG will continue to provide input and assistance to the ETA in an effort to ensure 
the integrity of the reported data. 

ETA’s detailed response and auditor’s conclusions are included at the end of each 
finding. In addition, the Deputy Assistant Secretary’s complete response is included as 
Appendix III. 
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BACKGROUND 

At the beginning of Fiscal Year (FY) 1999, the Secretary of Labor included youth services 
as one of her top priorities. At that time, youth training and related services under the 
Department of Labor’s (DOL) auspices were provided under the Job Training 
Partnership Act (JTPA); yet, Public Law 105-220, the Workforce Investment Act (WIA), 
signed August 7, 1998, had already been enacted to replace the JTPA program effective 
July 1, 2000. With the imminent demise of the JTPA program and the beginning of the 
WIA program with its heavy emphasis on serving youth, the Office of Inspector General 
(OIG) decided to conduct a performance audit of the JTPA, Title II-C, Out-of-School 
(OSY) program. 

We performed this audit of the JTPA OSY program because even though JTPA was 
being phased out, the WIA still includes youth programs and still requires performance 
reporting. The OIG had not previously conducted a performance audit of the JTPA 
OSY program and believed that the audit results could be beneficial to DOL 
management in assessing where the WIA program could make improvements in 
administering and reporting on the WIA youth programs. 

Beginning July 1, 2000, adult and youth training programs administered by the 
Department under JTPA Titles II and III changed to WIA Title I programs. The 
purposes of the WIA Title I programs are identified in Subtitle A, Section 106, which 
established the WIA program’s primary objectives. Program managers use these 
objectives to establish the new program’s goals and performance measures. As stated in 
the WIA: 



The purpose of this subtitle is to provide workforce investment 
activities, through statewide and local workforce investment 
systems, that increase the employment, retention, and earnings 
of participants, and increase occupational skill attainment by 
participants, and, as a result, improve the quality of the 
workforce . ... 

Likewise, the JTPA, as amended, stated that one of its purposes was to: 

. . . establish programs to prepare youth . . . facing serious 
barriers to employment for participation in the labor force by 
providing job training and other services that will result in 
increased employment and earnings, increased educational and 
occupational skills, and decreased welfare dependency. . . . 

JTPA, Title II, Part C, provided year-round training and employment programs for youth 
- both in-school and out-of-school - to improve their long-term employability, skills, or 
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transition to other education or training. We limited our audit to the Title II-C OSY 
component, which is required by the Act to serve at least 50 percent of the total Title II-C 
program participants in each service delivery area. 

An out-of-school youth was defined at Title 20 Code of Federal Regulations (CFR) , Part 
628.803: 

(a) An out of school youth is a youth who does not meet the 
definition of in-school youth as set forth in paragraph (b) of this 
section. An out-of-school youth shall be eligible to participate in 
programs under this subpart, if such individual is: 

(1) Age 16 through 21, and 

(2) Economically disadvantaged. 

(b) . . . In-school youth means a youth who has 
not yet attained a high school diploma and is 
attending school full time. 

20 CFR 628.803 (h) (2) further defined out-of-school youth as a youth who has attained a 
high school diploma or an equivalency, is habitually truant or is attending an alternative 
school program. Such programs may be operated either within or outside of the local 
public school system and can offer either a high school diploma or equivalency. 

With WIA’s major emphasis on performance outcomes and the Government 
Performance and Results Act of 1993 (GPRA) requirement for Federal agencies to 
annually report their programs’ outcomes, the validity and veracity of future WIA 
performance outcomes data is not just a necessity, but a requirement. 

WIA regulation 20 CFR, Part 667.300(e) provides: 

Annual Performance Progress Report. An annual performance 
progress report for each of the three programs under title I, 
subpart B is required by WIA sec. 136(d). 



(2) States submitting annual performance progress reports that 
cannot be validated or verified as accurately counting and 
reporting activities in accordance with the reporting instructions, 
may be treated as failing to submit annual reports, and be subject 
to sanction [Emphasis added.] 

The GPRA requires program managers to establish performance goals and systems to 
verify and validate the numbers reported. The GPRA states: 
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(а) In carrying out the provisions of section 1105(a) (29), the 
Director of the Office of Management and Budget shall require 
each agency to prepare an annual performance plan covering 
each program activity set forth in the budget of such agency. 

Such plan shall- 

(1) establish performance goals to define the level 
of performance to be achieved by a program 

activity; 

(2) express such goals in an objective, 
quantifiable, and measurable form unless 
authorized to be in an alternative form under 
subsection (b); 

(3) briefly describe the operational processes, skills and 
technology, and the human, capital, information, or other 
resources required to meet the performance goals; 

(4) establish performance indicators to be used in measuring or 
assessing the relevant outputs, service levels, and outcomes of 
each program activity; 

(5) provide a basis for comparing actual program results with the 
established performance goals; and 

(б) describe the means to be used to verify and validate 
measured values. [Emphasis added.] 

As demonstrated from these WIA and GPRA authorities, the veracity of reported 
performance outcomes is imperative. 

Performance measures for the Title II-C Youth Program 

In addition to the adult standards in the JTPA, Section 106(b)(3) (e.g., placement in 
unsubsidized employment), Section 106(b) (4) provided additional factors for youth 
standards. In summary, per Standardized Program Information Report (SPIR) 
instructions, positive outcomes used in calculating performance measures for OSY 
included: 

• placement in unsubsidized employment — minimum 20 hours per week, or 

• attained at least one of the following four employability enhancements: 
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1. Youth employment competency (must complete two of three 
competencies defined by the Private Industry Council (PIC)). 

2. Return to full-time secondary school. 

3. Completed major educational level -- secondary or post-secondary. 

4. Entered non-JTPA Title II training. 

See appendix I for the Primary Criteria for Youth Employability Enhancements. 
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OBJECTIVES, SCOPE, AND METHODOLOGY 



Our audit objectives were to determine: 

• whether out-of-school youth participants’ reported outcomes for positive 
terminations were accurate and fully documented (see finding 1); and 

• what impact program interventions had on participants’ post-program 
earnings (see finding 2). 

Our audit universe included youth who: 

• received a positive termination from the program — either entered 
unsubsidized employment or attained an employability 
enhancement — between July 1, 1997, through June 30, 1998, and 

• were identified in SPIR as out-of-school at the time of enrollment. 

Using PY 97 SPIR data, we identified an audit universe of 35,323 participants who met 
these audit criteria. 

We audited program services and outcomes for 16 randomly-selected youth at each of 34 
randomly-selected Service Delivery Areas (SDA) - a sample size of 544 participants. 
However, not all selected SDAs had 16 participants who met the out-of-school criterion; 
therefore, the audit sample was reduced to 510 out-of-school youth participants. As a 
result of our audit work, we identified 1 1 of the 510 sampled participants who were really 
in-school youth, reducing the final sample to 499 participants. 

Our audit procedures included, but were not limited to: 

• comparing selected SPIR data (e.g., services received, reported outcomes, 
etc.) with information obtained from reviewing the SDAs’ participants files, 

• reviewing participants' files for documentation of services and outcomes 
reported (including types of training provided to participants) , 

• interviewing SDAs’ staff regarding participant file information, 

• evaluating and comparing SPIR definitions to states’ and SDAs’ 
definitions, 
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• reviewing states’ wage files or obtaining employers’ confirmations to 
confirm reported entered unsubsidized employment outcomes, and 

• analyzing and comparing training activities to reported outcomes and 
participants’ earnings for 24 months (8 quarters) after termination from the 
program. 

We conducted our audit in accordance with Government Auditing Standards for 
performance audits issued by the Comptroller General of the United States. 

We began our fieldwork on August 4, 1999, and completed it on April 4, 2000. 

Audit procedures used to complete the review were not designed or intended to give an 
opinion on the adequacy of internal or program controls covering performance reporting 
at the local, state, or Federal level. 
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FINDINGS AND RECOMMENDATIONS 



1. Documented Reported Positive Program Outcomes for the JTPA Title II-C Out-of- 
School Youth Program Were Significantly Lower Than Those Reported. 

Our audit of the PY 1997 JTPA Title II-C OSY program revealed that positive program 
outcomes that were sufficiently documented were significantly lower than those outcomes 
reported on the SPIR and in the Chief Financial Officer (CFO) Report. Our 
determination that some outcomes were not adequately documented is not intended to 
imply that only the documented outcomes are valid positive outcomes. Some of the 
outcomes that were not documented may have, in fact, been attained. However, unless a 
reported outcome is documented, the outcome’s validity is not certain. 

Reported positive outcomes for 46 percent of the participant sample (228 of 499) we 
evaluated were not substantiated by evidence; i.e., the outcomes were either invalid or 
not properly documented. While we found this significant rate of undocumented 
outcomes, we are not implying that ETA has misrepresented youth program outcomes; 
ETA reports what the states report to ETA. While some of the reported outcomes in 
question may be the result of poor record keeping, other questionable outcomes were, in 
fact, invalid outcomes. 

It is also not the intent of this report to damage either the WIA program reforms ETA is 
attempting to implement or future youth programs. However, it is intended to alert ETA 
to a very serious problem that must be fixed to ensure that only documented WIA 
program outcomes are reported. Otherwise, the WIA program’s success will never be 
really evaluated. ETA has indicated it takes this issue very seriously and will work to 
improve the documentation issue. 

Inadequate documentation of positive program termination status 

The 499 sampled terminations we examined included 401 participants who were reported 
as entering unsubsidized employment (EMP); however, only 258 of the 401 EMP 
terminations (64 percent) met the positive outcome reporting criteria. (See part A of this 
finding.) The remaining 98 terminations were participants reported as having attained a 
youth employability enhancement. Only 13 of these 98 (13 percent) participants' files had 
adequate documentation to support the enhancement termination. (See part B of this 
finding.) 

Inadequate documentation of overall employability enhancements 

ETA not only reports the youth program’s overall positive terminations but also provides 
outcomes statistics by outcome type - entered unsubsidized employment or obtained 
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employability enhancement (s). Therefore, SDAs can report one or more youth 
employability enhancements for participants regardless of their termination status; i.e., a 
participant who was reported for overall outcomes reporting as having entered 
unsubsidized employment could also have one or more SPIR-reported employability 
enhancements. 

For the 499 sampled participants, the SPIR reported 308 employability enhancements for 
283 participants. Our evaluation revealed that only 27 percent of both individuals (75 of 
283) with reported employability enhancements and total employability enhancements 
(84 of 308) were adequately documented. (See part B of this finding.) 

The OSY program we audited was operated under the JTPA which has now been 
replaced by the WIA. Regardless of the enabling legislation, JTPA or WIA, youth 
programs are ongoing and performance outcomes will continue to be reported against 
established standards. The question to be answered is whether the reported outcomes 
can be validated and relied upon by ETA for GPRA reporting. 

The JTPA, Section 165 provided: 

(a) (1) Recipients shall keep records that are sufficient to permit the 
preparation of reports required by this Act. . . . 



(c) Each State, each administrative entity, and each recipient . . . receiving 
funds under this Act shall - 



(2) prescribe and maintain comparable management information systems . . . 
designed to facilitate the uniform compilation, cross tabulation, and analysis of 
programmatic, participant . . . data . ... 



(d) (1) The reports required . . . shall include information pertaining to .. . 

(B) the activities in which participants are enrolled, and the 
length of time that participants are engaged in such activities; 

(C) program outcomes .. .for participants. . . . 

(e) The Governor shall ensure that requirements are established for retention 
of all records pertinent to all grants awarded . . . including . . . participant 
records and supporting documentation. . . . [Emphasis added.] 
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WIA regulation 20 CFR, Part 667.300(e) provides: 

Annual Performance Progress Report An annual performance progress report 
for each of the three programs under title I, subpart B is required by WIA sec. 
136(d). 



(2) States submitting annual performance progress reports that cannot 
be validated or verified as accurately counting and reporting activities in 
accordance with the reporting instructions, may be treated as failing to submit 
annual reports, and be subject to sanctions. . . . [Emphasis added.] 

The GPRA requires that program managers establish performance goals and systems to 
verify and validate the numbers reported. The ETA will be using state reported 
outcomes to report the outcomes of the WIA program for GPRA purposes. The GPRA 
states: 



(a) In carrying out the provisions of section 1105(a) (29), the Director of the 
Office of Management and Budget shall require each agency to prepare an 
annual performance plan covering each program activity set forth in the budget 
of such agency. Such plan shall- 

(1) establish performance goals to define the level of 
performance to be achieved by a program activity . . . 



(4) establish performance indicators to be used in measuring or 
assessing the relevant outputs, service levels, and outcomes of 
each program activity; 

(5) provide a basis for comparing actual program results with the 
established performance goals; and 

(6) describe the means to be used to verify and validate 
measured values. [Emphasis added.] 

As these criteria demonstrate, JTPA was specific that activities and outcomes should be 
documented, and both the WIA program regulations and the GPRA emphasize the 
necessity of having performance outcomes data that can be validated/verified. We found 
reported outcomes for the JTPA Title II-C OSY program did not meet this validation/ 
verification standard. 

Table 1.1 summarizes the overall audit results for reported performance outcomes 
terminations and youth employability enhancements. 



Office of Inspector General 



12 



JTPA ’s Title II-C Out-of-School Youth Performance Audit 



Table 1.1 

Comparison of Reported Versus Verified Outcomes 


Outcomes Reported 


Reporte 

d 


Verifie 

d 


Percen 

t 


Entered Unsubsidized Employment Terminations 
(performance related) - See finding part A. 


401 


258 


64.3% 


Youth Employability Enhancement Terminations 
(performance related) - See finding part B. 


98 


13 


13.3% 


Total Employability Enhancements (including 98 
that 

were performance related) - See finding part B. 


283 


75 


26.5% 



A. A significant number of entered unsubsidized employment terminations 
were reported in error and others are in question. 

Of the 401 participants in our sample who were reported in the SPIR as entering 
unsubsidized employment, only 258 (64 percent) reported placements were verified as a 
program outcome. As table 1.2 on the following page shows, we validated 238 
placements for participants who received services beyond objective assessment. We were 
able to verify an additional 20 placements, but we were unable to verify that these 20 
enrollees participated in a valid program activity even though counselors’ notes or 
management information system (MIS) forms indicated there may have participated 
beyond objective assessment. 

In our opinion, the remaining 143 reported entered unsubsidized employment outcomes 
were either not properly reported or are questionable for outcomes reporting based on 
our review of participants’ case files data, states’ wage records, and employer contacts. 
Table 1.2 on the following page identifies the specific reasons why the OIG questions 
these reported outcomes. 
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Table 1.2 

Analysis of 401 Participants Reported As Entered Unsubsidized Employment 


Placements Verified 


Placements verified 


238 


59.4% 


Placements verified but inadequate support for program activity 


20 


5.0% 


Total verified placements 


258 


64.3% 


Placements Improperly Reported to ETA 


No state wage record - employer confirmed did not employ 


38 


9.5% 


MIS reporting error, no placement recorded in participant file 


1 


0.2% 


Participants received no services other than objective assessment 1 


21 


5.2% 


Placements were beyond 90 days past last documented program 


36 


9.0% 


Total placements improperly reported 


96 


23.9% 


Placements in Question 






No state wage record - unable to contact reported employer 


32 


8.0% 


Worked for employer prior to or during JTPA enrollment 


15 


3.7% 


Total placements in question 


47 


11.7% 


Total placements improperly reported or in question 


143 


35.7% 



As shown above, 38 participants did not have wage records and employers confirmed 
with the OIG that they did not employ the individuals. In our opinion, failure to verify 
placements through two sources of information - wage records and employer 
confirmation - was significant evidence that the participants were not employed. 
Consequently, when both sources indicated no employment occurred, we considered the 
outcomes invalid. 

For the 32 participants with no wage records and for whom we were unable to contact 
the employers -- no response to phone calls, unable to locate, or did not return written 



1 We did not include in our audit universe or sample any participants which the SP1R showed as receiving only 
objective assessment, regardless if the termination status was “entered unsubsidized employment.” In computing performance 
outcomes, ETA does not include such individuals in its computations since they received no other services. The individuals 
included in this category of “placements improperly reported to ETA” were included in ETA’s outcomes analysis because the 
SPIR reported activities for them other than objective assessment and also reported that they entered unsubsidized 
employment. We found no evidence in the participants’ files to indicate they received any services other than objective 
assessment. 
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confirmations — it is possible that some of these participants worked for the employers 
but for whatever reason the employers failed to report wages. However, with no wage 
records for these 32 participants and our not being able to contact the employers, in our 
opinion, these outcomes are in question. 

For the 21 participants we indicated received only objective assessment, we did an 
exhaustive review of each participant’s file, and there was no indication in these files that 
any services were received beyond objective assessment. 

For the 36 participants for whom entered unsubsidized employment outcomes were 
reported more than 90 days after last activity, in our opinion, these cases were not simply 
record keeping issues where SDAs may not have been diligent about documenting 
program activities and services. The 90-day rule is in place to attempt to relate post- 
program employment to JTPA services. It is common practice among SDAs — and has 
been reported in other audit reports on the JTPA program — to keep participants 
enrolled in the program for long periods of inactivity until they obtain employment, then 
terminate them and report an entered unsubsidized employment outcome. 

Regarding the 1 5 participants who worked for the employer prior to or during JTPA 
enrollment, we took prior employment information from the participants’ own files, not 
from wage records. The employment at issue had nothing to do with JTPA involvement. 



B. Documentation supporting youth employability enhancements was 
seriously deficient. 

We reviewed documentation for 308 youth employability enhancements reported for 283 
of the 499 sampled participants regardless of the type of positive termination reported for 
the participants. We found only 27 percent of the enhancements to be adequately 
documented. 

Although only 98 participants’ termination status was reported as having attained an 
employability enhancement, these 98 participants accounted for 105 reported 
employability enhancements. An additional 185 participants whose termination status 
was reported as having entered unsubsidized employment also accounted for an 
additional 203 reported employability enhancements. 

Table 1.3, on the following page, shows that the documentation to support reported 
employment enhancements was seriously deficient. 
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Table 1.3 

Undocumented Youth Employability Enhancement Terminations and 
Youth Enhancements Claimed for Participants Entering Employment 


Reportable 2 
Enhancements 


Reported Enhancement 
Terminations (£&) 


Enhancements Attained 
Regardless of Termination 


Reported 


Undoc. 


Percent 


Reported 


Undoc. 


Percent 


Youth Employment Competency 


68 


57 


83.8% 


203 


160 


78.8% 


Major Education Level 


28 


23 


82.1% 


91 


51 


56.0% 


Returned to Full-time School 


5 


5 


100.0% 


5 


5 


100.0% 


Entered Non-Title II Training 


4 


4 


100.0% 


9 


8 


88.9% 


Total Enhancements Reported 


105 


89 


84.8% 


308 


224 


72.7% 



For the 98 participants whose termination status was attained employability 
enhancement, only 16 of the 105 (15.2 percent) reported employability enhancements 
were adequately documented. For all 283 3 participants for whom employability 
enhancements were reported, only 84 of 308 (27.3 percent) of the enhancements were 
documented. The following discussions relate to the four specific reportable 
enhancements shown in the previous 
table 1.3. 



1) Youth Employment Competency (YEC) Enhancement 

Of the 203 YEC enhancements reported, 160 (78.8 percent) were not adequately 
documented in accordance with ETA’s reporting instructions. 

To receive credit for the YEC enhancement, the participant must successfully complete 
two of the three following competencies in which they were deficient at the time of 
enrollment: 

a. Pre-Employment/Work Maturity (PEWM) , 

b. Basic [Education] Skills Training (BST), or 



Participants may attain more than one enhancement while enrolled in the program. 

includes the 98 participants for whom “attained employability enhancement” was the reported termination status. 
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c. Occupational Skill Training (OST) . 

The SPIR Format, Appendix A, page 52, provided the criteria used during the audit to 
evaluate the completion of the YEC activity. 

Competency gains must be achieved through program participation and be 
tracked through sufficiently developed systems that must include: quantifiable 
learning objectives, related curricula/training modules, pre- and post- 
assessment, employability planning, documentation, and certification. 

[Emphasis added.] 

Pages 46 and 48, paragraph B6, of the same document cited above, contained the 
following documentation requirement: 

A sufficiently developed youth employment competency system must include the 
following structural and procedural elements: 

6. Documentation 

* Maintenance of participant records and necessary reporting of competency- 
based outcomes to document intra-program learning gains achieved by youth. 

Specific PEWM competency requirements were identified on page 49 of the document. 

In order for an attainment to be reported in the area of pre-employment work 
maturity, at least one PIC-certified competency statement must be 
developed/quantified in each of ... 11 core competencies - provided that at 
least 5 of these learning objectives were achieved during program 
intervention. . . . 

Table 1.4 on the following page shows the combinations of how the grantees reported the 
YEC enhancements to the DOL. 
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Table 1.4 



Youth Employability Competency Enhancements Reported Outcomes 
Compared to Enhancements Documented in Participants’ Files 



YEC Enhancement Combinations 


Reporte 

d 

Per 

SPIR 


Documente 

d 

In Files 
Per Audit 


Undocumented 


Number 


Percent 


PEWM/BST/OST 


24 


1 


23 


95.8% 


PEWM/BST 


59 


16 


43 


72.9% 


PEWM/OST 


106 


22 


84 


79.2% 


Total Competencies Which 
Included PEWM as Part of YEC 


189 


39 


150 


79.4% 


BST/OST 


14 


4 


10 


71.4% 


Overall YEC Results 


203 


43 


160 


78.8% 



YEC enhancements represented 66 percent (203 of 308) of the reported OSY 
employability enhancements. As demonstrated in table 1.4, by far (189 of 203) the most 
reported YEC enhancements included the PEWM competency component. The 
remaining employability enhancements were the combinations of BST and OST. 

As table 1.4 shows, the YEC enhancement which included the PEWM component 
accounted for 150 of the 160 questionable YEC enhancements. The PEWM competency 
was the questionable component in 143 of the 150 questionable reported YEC 
enhancements that contained the PEWM competency. 

• Nine participants’ files (6.0 percent) indicated the participant had passed 
the pre-test; therefore, the participant should not have been enrolled in 
PEWM. 

• Eleven participants’ files (7.3 percent) had no evidence that a pre-test was 
ever given to the participant. 

• Fifteen participants (10.0 percent) did not successfully complete the 
PEWM even though successful completion was reported on the SPIR. In 4 
of these 15 cases, the auditee’s MIS did not report PEWM as an activity. 
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• Fifty-five participants’ files (36.7 percent) contained inadequate 
documentation to support the successful completion of the PEWM 
intervention. In 29 of these 55 cases, the auditee’s MIS did not report 
PEWM as an activity. 

• Fifty participants’ files (33.3 percent) had no evidence to support either the 
participants’ enrollment in, or completion of, the PEWM intervention. In 
30 of these 50 files, the auditee’s MIS did not report PEWM as an activity. 

• Three participants’ files (2.0 percent) indicated the participants were not in 

JTPA Title II-C activities. 

Another seven participants’ files showed the PEWM outcome was a successful 
completion; however, the YEC enhancements were questionable because the BST and/or 
OST segments of the YEC enhancement were not supported. 

While the JTPA requirements and complexity of documenting the PEWM competency 
may lead to documentation and record keeping deficiencies, in our opinion, the problem 
with the PEWM competency goes beyond just documentation issues. As previously 
stated, we evaluated the attainment of the PEWM competency based on ETA’s 
requirements. 



2) Completed Mqjor Level of Education (MLE) Enhancement 

To report an MLE employability enhancement, the SDA was required to document that 
the participant actively participated in JTPA for 90 days or 200 hours prior to completion 
and successfully completed, as a result of JTPA activity, one of the two following criteria: 

a. must have earned a general equivalency diploma (GED) or high school 
(HS) diploma or equivalent at the secondary level, or 

b. obtained a written certification of post-secondary level completion. 

Out of 91 reported MLE enhancements for our participant sample, 51 (56 percent) were 
not supported. We found: 

• Twenty-one participants’ files (23.1 percent) did not document fulfilling 
the time requirement of 90 days or 200 hours of active JTPA participation 
prior to completion. 

• Six participants’ files (6.6 percent) did not complete the MLE activity in 
which they were enrolled. Additionally, four of these cases did not fulfill 
the time requirements. 
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• Seven participants’ files (7.7 percent) had no evidence to support 
enrollment in an MLE activity. In one of these cases, the auditee’s MIS 
did not report MLE as an activity. 

• Seventeen participants’ files (18.7 percent) did not include enough 
documentation to determine attainment of the MLE youth enhancement. 

The following two examples demonstrate invalid MLE enhancements that were reported 
as positive outcomes. 

Example 1 . Participant was a 20-year old high school graduate assessed with a sixth 
grade reading skill level and a fourth grade math skill level. He was enrolled in JTPA 
Title II-C basic skills training for approximately 4 months, Title II-B summer youth work 
experience program performing janitorial work for 2 months, and a Title II-C computer 
class for 1 month. 

He completed the computer class unsatisfactorily. Consequently, the instructor told him 
to either repeat the class or re-enroll in basic skills training. The participant did not 
comply and was subsequently terminated because of his lack of interest and failure to 
maintain contact with the program. 

Yet, he was erroneously reported as having attained the youth employability 
enhancement of “achieved a major level of education,” a positive youth enhancement 
termination. This participant did not satisfactorily complete his training, nor did he (1) 
earn a GED/HS diploma or equivalent at the secondary level, or (2) obtain a written 
certification of completion at the post-secondary level, a requirement to claim the MLE 
enhancement. 

Example 2. The SDA reported this participant as having “achieved a major level of 
education.” His SPIR records also indicated the participant completed the basic 
education skills component of the youth employment competency. 

The participant’s file indicated he completed 100 hours training in “other employment 
skills,” but file documentation was not adequate to verify training or completion. The file 
contains no reference to his attaining basic education skills or completing a major level of 
education. SDA personnel agreed that the participant did not participate in these 
activities and indicated these outcomes were improperly checked on the forms. 

While there is no documentation standard to which SDAs are held, there are standards 
that must be met to report an MLE outcome. Unless the attainment of that standard is 
documented, there is no assurance the outcome was attained. 
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3) Returned to Full-Time School (RTS) Enhancement 

In order to report the RTS employability enhancement, SDAs were required to 
document that the following criteria were met: 

a. the youth returned to full-time secondary school, if, at time of intake, youth 
was not attending school, exclusive of summer, and the youth had no high 
school diploma or GED, and 

b. prior to JTPA termination, the youth had been retained in school one 
semester or 120 calendar days prior to termination, and the youth was 
making satisfactory progress in school. 

None of the five RTS youth enhancements reported met the documentation 
requirements for the following reasons: 

• three participants’ files had insufficient documentation to support the 
participant: (1) returned to full-time school, and/or (2) was attending 
school during JTPA enrollment for 120 days or one semester, and/or (3) 
had obtained a PIC approved YEC in BST or OST ; and 

• two participants did not return to full-time school. 

4) Entered Non-Title II Training (ENT) Enhancement 
The attainment of the ENT youth employability enhancement required that: 

a. prior to termination, the participant entered occupational training not 
funded by JTPA Title II which builds upon Title II training, 

b. the non-Title II training did not duplicate Title II training, and 

c. the participant was retained in non-Title II training for 90 days or 200 
hours prior to completion of II-C activities or received a certification of 
occupational skill attainment. 

Eight of the nine ENT youth employability enhancements were questionable. In sue 
cases, there was no evidence in the participants’ case files to support the participants’ 
enrollment in non-Title II activities. Some documentation existed for another participant 
but the support was inadequate to document the enhancement. Finally, another 
participant did not successfully fulfill the time requirement of 90 days or 200 hours of 
active non-Title II training prior to completing the JTPA activity. 
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Overall effect of our audit results 

In our opinion, based on the significance of the rates of positive outcomes that were not 
documented, the OSY outcomes rates reported by the SPIR and included in the PY 1997 
CFO Report were significantly lower than reported. While some of the undocumented 
outcomes may have, in fact, been attained, unless a reported outcome is documented, the 
outcome’s veracity is not certain.. However, we are certain the outcomes rates were not 
as high as reported, because many of the outcomes we classified as undocumented were, 
in fact, invalid based on evidence we reviewed. 

The JTPA Title II-C OSY program’s true effectiveness is probably somewhere between 
ETA’s reported rates and the OIG’s documented successes; i.e., positive outcomes are 
lower than ETA’s reported, and some of the outcomes the OIG found not to be 
documented were no doubt attained. However, unless the outcomes are documented, 
the true effectiveness of youth training programs cannot be determined. 

To meet both WIA program regulations and GPRA requirements that performance 
outcomes must be able to be validated/verified, significant improvement must be made by 
all parties - the service deliverers, Workforce Investment Boards, states, and the ETA -- 
involved in the current WIA program to ensure that reported performance outcomes are 
accurate for current and future youth programs’ outcomes reporting. 

We recommend the Assistant Secretary for Employment and Training: 

• Notify all states and substate WIA grantees of the audit results and 
emphasize the necessity for states and their grantees to validate outcomes 
data as part of their monitoring programs and training provider eligibility 
determinations. 

• Emphasize to the states and substate grantees the importance of 
documenting not only reported outcomes but also specific services 
provided, dates services were provided, and actual program exit date - 
date of last activity. 

• Include in ETA’s monitoring program a review of the states’ and substate 
WIA grantees’ implementing such a data validation effort. 

ETA’s Response to Our Draft Report: 

With regard to our specific recommendations, the Deputy Assistant Secretary for 
Employment and Training stated: 
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. . . ETA will issue a program guidance letter informing the States of the 
Endings and recommendations contained in this report. States will be asked to 
inform their WIA substate grantees of the audit findings. . . . [ETA] will 
encourage States and their grantees to include outcome data validation as part 
of their program reviews. . . . 

.. .the report does not clearly identify the standards of documentation that were 
applied. It would be helpful if the report could do so and could acknowledge 
the extent to which these standards are consistent with previous guidance issued 
under either JTPA or WIA. . . . 

. . . Under WIA regulations, monitoring of subrecipient or substate activities is 
primarily a State responsibility. 

At the federal level, ETA 's regional staff have primary responsibility for ongoing 
program oversight and monitoring. We will convey the findings of the report to our 
regional staff and instruct them to work with their States in implementing data 
validation efforts as part of the States' oversight of their WIA substate grantees. We 
will also instruct regional staff to include this area in any program reviews at the State 

level. 

In ETA’s additional comments, the Deputy Assistant Secretary stated: 

The report states that 38 terminees were found to have no wage records and the 
"employer confirmed (they) did not employ (the) participant. " ... We 
recommend that the report explain the steps taken to verify employment 
outcomes and acknowledge the difficulty of verifying employment outcomes not 
reflected in the wage record system some two years after they are reported. 

Unfortunately the solution to documentation and verification issues is not as 
simple as issuing more explicit guidelines or conducting additional data 
validation reviews. We are mindful that a core principle of the reforms under 
WIA is to improve services to youth. We also need to acknowledge the 
administrative burden imposed on the programs to collect extensive 
documentation. ... It could be counterproductive to the purposes of WIA if 
administrative requirements become so burdensome that they are a deterrent to 
the delivery of program services. 

Auditor’s Conclusion: 

We agree with ETA’s plan to get the audit findings out to the states. With respect to 
ETA’s plan to encourage states to include outcomes’ data validation as part of their 
monitoring programs, we believe ETA needs to take stronger action than simply 
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encouraging the states to validate data, since both WIA and GPRA requires that 
reported outcomes be verifiable. 

ETA’s response raised the issue of what documentation standard the OIG used in this 
audit. We used ETA’s SPIR reporting requirements and the SDAs’ instructions to their 
staff and service providers through their annual plans or service provider agreements. 

The majority of the SDAs utilized the SPIR’s reporting requirements as the basis for their 
reporting; however, in many cases, SDAs supplemented the SPIR requirements with 
their own. As a result, we cannot provide all the criteria used to evaluate the adequacy of 
the documentation. However, we reviewed each participant’s file and reported outcomes 
using SDA-furnished criteria and SPIR instructions, and found documentation of 
participation to be a significant problem no matter what criteria we applied. 

With respect to the Deputy Assistant Secretary’s response to our recommendation 
regarding the scope of ETA’s monitoring, we recognize that, as ETA points out, the 
monitoring of sub recipient or subspace activities is primarily a State responsibility under 
the WIA regulations. However, the Secretary is authorized to monitor all recipients and 
subrecipients. Further, DOL’s uniform administrative requirements, found at 29 CFR 
Parts 95 and 97, require grantees to monitor subgrants, and Part 97 (which applies to 
governmental entities, including states) specifically requires grantees to monitor 
subgrants to assure performance goals are being achieved. As only valid outcomes count 
toward the determination of goal achievement, we believe such outcomes must be 
verified by the states. 

ETA’s additional comments requested further information on our procedures to verify 
employment. We obtained wage records for all sampled participants. We reviewed the 
wage records to verify the reported employment. If the employer appeared on the 
participant’s wage record, we considered the placement valid. If the employer was not on 
the participant’s wage record but the participant had wage records for any other 
employer in the termination quarter, we considered the reported “entered unsubsidized 
employment” outcome to be valid. In some cases, SDAs had written employer 
verifications, which we accepted. 

For participants whose wage records did not verify employment, we attempted to contact 
the employer by phone. If we were successful, we talked to the employer’s human 
resource officer or the owner. If the employer verbally confirmed the employment, we 
considered it to be valid. If the employer stated the participant was not hired, we 
considered the reported employment invalid. As stated in the report finding, in our 
opinion, failure to verify unsubsidized employment through two sources of information - 
wage records and employer confirmation - was significant evidence that the participants 
were not employed. When both sources indicated no employment occurred, we 
considered the outcomes invalid. 
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We disagree with ETA’s suggestion that we are calling for extensive documentation, 
which ETA believes could burden both the programs and employers to the point of 
becoming counterproductive and a deterrent to service delivery. We believe the burden 
would have been minimal, and subsequent validation more easily accomplished, if the 
SDAs had: obtained adequate documentation (from the participant, school and/or 
employer) at the time an outcome occurred; maintained such documentation in the 
participant file; and reported only documented outcomes. 
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2. Participants’ Post-Program Earnings Were Affected by Program Interventions 
and Level of Participants’ Participation. 



We found that post-program earnings for youth who participated in OST activities were 
twice that of those who did not. In addition, for those OST participants who completed 
the OST activity, post-program earnings for the 2-year period after termination were 69 
percent higher than for those who did not complete the OST activity. 

Another audit objective was to determine whether the individual service strategy (ISS) 
was being used to provide training or services to specifically address participants’ 
employment barriers and to determine if the provided interventions impacted post- 
program earnings. Many identified barriers were not addressed because of the program’s 
limitations and because participants refused to participate, both issues outside the SDAs’ 
control. The SDAs did enroll the majority of participants in activities to address the 
barriers, but the participants did not complete approximately half the activities. 

A. Post-program earnings were directly impacted by the type of training a 

participant received. 

This audit revealed that post-program earnings for OST youth were twice that of those 
who did not participate in OST. Our 499 participant sample included 371 participants 
that were enrolled in at least one OST activity during their JTPA enrollment. Table 
2.1summarizes the results of our analysis of post-program earnings for all of the 499 
sampled participants who had earnings. 



Table 2.1 

Average Earnings for Individuals Based in Occupational Skills Training 



Quarters After Termination 



OST 


1 st Four Quarters 


2 nd Four Quarters 


2 Years Combined 


No. 


Quarterly 


Annual 


No. 


Quarterly 


Annual 


No. 


Quarterly 


Annual 


Yes 


321 


$2,787 


$9,291 


301 


$3,173 


$10,323 


340 


$2,904 


$17,910 


No 


101 


1,564 


4,818 


94 


1,651 


5,047 


112 


1,558 


8,580 


All 


422 


$2,494 


$8,220 


395 


$2,811 


$9,068 


452 


$2,570 


$15,599 



As stated earlier, the differences in earnings between those participating in OST and 
those who did not are significant. Both tables 2.1, above, and 2.2, on the next page, show 
higher earnings for OST participants. 
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Table 2.2 shows that, regardless of the combination of services a participant receives, 
OST participants fare better than non-OST participants. 



Table 2.2 

Occupational Skill Training’s Impact on Participants’ Earnings 








Participant Averages 


Training Received 


1 st Four Quarters After 
Termination (i a Year) 


2 nd Four Quarters After 
Termination (2" d Year) 


OS 

T 


BS 

T 


JS/JR 


No. 


Quarterly 

Earnings 


Annualize 

d 

Earnings 


No. 


Quarterly 

Earnings 


Annualize 

d 

Earnings 




Yes 


Yes 


44 


$2,223 


$7,160 


43 


j $2,741 


$9,388 


Yes ; 


No 


33 


2,641 


8,630 


34 


3,075 


9,649 


No 


Yes 


104 


2,962 


9,678 


92 


3,351 


11,087 ^ 




No 


140 


2,868 


9,828 


132 


3,216 


10,269 




Yes i 


Yes 


44 


1,396 


4,176 


42 


1,147 


3,011 


No 


No 


35 


j 1,547 


4,819 


35 


1,916 


6,416 


No 


Yes 


21 


[ 1,838 


5,842 


16 


2,259 


6,760 




No 


! 4 


| 3,835 


11,506 


i 


] 3,810 


15,240 



Table 2.2 shows the combination of training activities participants can be enrolled in while 
in JTPA. As highlighted in the table, a participant receiving OST and JS/JR training 
earned on average $9,678 for the first four quarters after termination while the 
participant enrolled in BST and JS/JR earned only $4,176. The gap between these two 
groups is much greater for quarters five through eight after termination; the OST and 
JS/JR groups earnings increased 15 percent to $1 1,087 while the BST and JS/JR groups 
earnings decreased 28 percent to $3,011. 

We also evaluated the post-program earnings impact of participants completing the OST 
activity. For those OST participants who completed the activity, we estimate post- 
program earnings for the 2 -year period after termination to be 69 percent higher than for 
those who did not complete the activity — $26,098 to $15,454. 



4 Participant was a 20-year old high school graduate who received no training or services during JTPA enrollment. 
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8. Almost half of the training activities to address participants’ barriers were 
not completed. 

The purposes of the objective assessment and the ISS are to identify a participant’s 
barriers and then create a strategy to address those barriers. 

20 CFR 628.515 requires an objective assessment be performed on each participant. 
Objective assessment, per 20 CFR 628.515(b), means an examination of the capabilities, 
needs, and vocational potential of a participant and is to be used to develop an individual 
service strategy and employment goal. ... 20 CFR 628.520 indicates that an ISS will be 
developed for each participant as a cooperative effort between the SD A/service provider 
and the participant. 

We estimate: 

• 98.6 percent had an ISS of which: 

• 86.7 percent had an employment goal. 

• 87.3 percent had appropriate achievable objectives. 

• 86.2 percent had intervention plans to address the participants’ 
barriers. 

We attempted to compare ISS-identified employment barriers to the training and services 
participants received. We combined training activities into the following three categories: 
(1) basic skills, (2) occupational skills, and (3) job-seeking/job-retention (job- 
related/readiness) skills. Each of these categories is defined as follows: 



Definition of Training Categories Used in this Report 

Basic Skills Training (BST) includes: basic skills, GED preparation, remedial education, 
and English as a second language. 

Occupational Skills Training (OST) includes: on-the-job training, work experience, 
cooperative education, limited-internships, college/Jr. college courses, and 
vocational/technical school. 

lob-seeking/ Tob-retention Skills Training (JS/JR) includes: preemployment/work 
maturity, job clubs/job search assistance, coaching/counseling, and other seminars and 
workshops provided by the SDAs or their service providers. 
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Although participants had many types of barriers identified, we concentrated on the 
impact JTPA training had on four barriers. 

• basic skills deficiency, 

• high school dropouts, 

• lacks work experience/training, and 

• lacks job-seeking and/or job-retention skills. 

Table 2.3 presents the primary interventions participants in our sample received to 
address their barriers. Because the majority of the participants have more than one 
barrier, the individual totals will not add to the number of participants in the sample. 



Table 2.3 

Barriers and Interventions 


Barriers 


No. 


Interventions 


BST 


OST 


JS/JR 


Enrolle 

d 


% 


Enrolle 

d 


% 


Enrolle 

d 


% 


Basic Skills Deficient 


289 


144 


49.8% 


193 


66.8% 


156 


54.0% 


School Dropout 


221 


152 


68.8% 


121 


54.8% 


139 


62.9% 


Lacks WE/OST 


309 


119 


38.5% 


237 


76.7% 


167 


54.0% 


Lacks JS/JR Skills 


156 


48 


30.8% 


139 


89.1% 


95 


60.9% 



As can be seen from the above table, the SDAs appeared to try to address participants’ 
barriers by enrolling them in related activities. For example, almost 50 percent of those 
with basic skills deficiencies and almost 70 percent of high school dropouts received basic 
skills training; almost 80 percent of those who lacked work experience or occupational 
skills received some occupational skills training; and 60 percent of those who lacked job 
seeking/job related skills received services in that area. 

However, enrollment in an activity did not ensure that participants received the necessary 
skills to overcome their barriers because many did not complete the activity. We found 
that 59 percent of the BST activities and 49 percent of the OST activities were not 
completed. 

We did not determine why participants did not complete the activities in which they were 
enrolled. However, one characteristic of high-risk youth is a continual state of flux, with 
major changes occurring in their lives or the paths they choose to take almost daily. While 
these high-risk youth may be hard to work with, it is evident that youth participants who 
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receive OST have higher earnings than those who do not. Also, those who complete the 
activity have higher earnings than those who do not. Consequently, to increase the 
potential for earning levels that will allow youth to be self-supporting, it is critical for the 
WIA youth program to find ways to keep youth actively participating in the program to 
completion. 

We recommend the Assistant Secretary for Employment and Training emphasize the 
importance of the WIA program’s not only enrolling youth in occupational skills training 
activities, but also finding ways to keep them actively participating in the program to 
completion, to maximize their post-program earnings. 

ETA’s Response to Our Draft Report: 

We will include this emphasis in the .. . program guidance letter disseminating 
the findings of the audit. We also issue program guidance information each 
year, typically about the summer employment opportunities program. This year 
we plan to issue comprehensive guidance on youth services under WIA and will 
include a section in this guidance letter on retention of participants. While we 
concur with the need to focus on retention strategies, it is also important to 
understand that these programs serve some of the most challenging youth. 

Many economically disadvantaged, out-of-school youth face multiple barriers to 
successful transition into the work force . ... 

Auditor’s Conclusion: 

We agree with the Assistant Secretary’s planned actions regarding this finding. 
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APPENDIX I 



PRIMARY CRITERIA FOR YOUTH EMPLOYABILITY ENHANCEMENTS 

In order to attain a youth employability enhancement, a participant must have 1) 
accomplished two of three PIC-defined youth competencies; 2) returned to full-time 
school; 3) achieved a major educational level; or 4) entered other non-Title II training. 
Each enhancement is discussed below: 

1. Youth competencies (must accomplish two of three PIC-defined) include: 

• pre-employment/work maturity skills, 

• basic academic skills, or 

• job specific skills. 



2. Return to full-time secondary school (e.g., junior high, middle, or high school ) . 

• Prior to JTPA termination the participant must have remained in school 
for one semester or at least 120 calendar days after becoming a JTPA 
participant. 

• SDAs must demonstrate that retention results from continuing, active 
participation in JTPA activities, and the youth must: 

be making satisfactory progress in school based on a written policy. 
[The policy should define both qualitative (e.g., performance on a 
criterion-referenced test or a grade point average) and quantitative 
(e.g., a time limit for completion of the program or course of study). 
Under mitigating circumstances, some participants not meeting the 
standards may be considered as making satisfactory progress during 
a probationary period.] (Per SPIR instructions.) 

for youth aged 16 through 21, attained either a PIC -approved basic 
academic skill or job specific skill competency; and 

for youth aged 14 through 15, attained either a PIC -approved basic 
academic skill or pre-employment/work maturity competency. 
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3. Completed major educational level -- both secondary and post secondary. 

• Must result primarily from active JTPA participation of at least 90 days or 
200 hours, usually prior to such completion, and 

for high school graduate or GED recipient, must be supported by a 
diploma or GED certificate or equivalent, and 

for post secondary accomplishment, requires a diploma or other 
written certification of completion at the post-secondary level. 

4. Entered non-JTPA Title II training. 

• Must have been retained in the non-JTPA program for at least 90-calendar 
days or 200 hours or must have received a certification of occupational skill 
attainment. 

• Must not have received JTPA Title II services while in non-Title II training. 
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APPENDIX II 
Sampling Plan 

The sampling plan for the audit was based on national populations of 1) SDAs and 2) 
JTPA Title II-C out-of-school youth program participants who met the following criteria: 

® received a positive termination from the program — either entered 
unsubsidized employment or attained an employability 
enhancement - between, and including, July 1, 1997, and June 30, 

1998, and 

@ the SPIR identified as out-of-school at the time of enrollment. 

The source of these populations was the PY 1997 SPIR computer file provided by Social 
Policy Research Associates (SPRA) . SPRA maintains the national SPIR system under 
contract with the U.S. Department of Labor, Employment and Training Administration. 

We used a two-stage sampling plan based on a two-sided 95 percent confidence level, 
expected error rate of 50 percent, and precision of ± 5 percent. First, DOL regions were 
combined into four geographic areas identified as stratum in the table below. Based on 
the number of SDAs in each stratum, a random sample of SDAs was selected. Second, a 
random sample of participants was selected for each sampled SDA. There were 634 
SDAs with 35,323 participants who met our audit criteria. The following table displays 
the resulting sampling plan. 



Stratum 


DOL 

Region 

s 


SDA Universe j 


J SDA Sample 


Part. 

Sample 

Size 


Total 

Part. 

Sampl 

e 


No. of 
Participants 
1-100 101 + 


All 


No. of 
Participants 
1-100 101+ 


All 


1 


1,2,3 


1 144 


12 


156 


6 


2 


8 


16 


128 


2 


5,7 


185 


26 


211 


1 

9 


2 


11 


16 


176 


3 1 


4, 6,8 


160 


11 


171 


7 


2 


9 


16 


144 


4 


9, 10 


80 


16 


96 


4 


2 


6 


16 


96 


Total | 




569 

E 


65 


634 


26 


8 


: 34 . 


: If 


544 : 



As shown in the prior table, we randomly selected a sample of 34 SDAs and 16 
participants for each SDA. The total sample size was 544 participants, however, as shown 
below, several of the SDAs did not have 16 participants that met the audit criteria. In 



Office of Inspector General 



34 



JTPA ’s Title II-C Out-of-School Youth Performance A udit 



those cases we reviewed all participants who met the audit criteria. Total participant files 
reviewed was 510. The sample is identified below. 



SDA Sample 

Name Size Name Size 



Western Arkansas Consortium 


16 


Lansing/Tri County E & T 


16 


Balance of Santa Clara County 


16 


Muskegon/Oceana Consortium, 


16 


Shasta County, CA 


16 


Central Mississippi SDA, MS 


16 


San Benito County, CA 


7 


Greater Lincoln Consortium, NE 


16 


Anaheim City in Orange County, 


3 


Union County, NJ 


14 


Jefferson County Consortium, CO 


16 


Ocean County, NJ 


16 


Pasco-Hernando JEP 16, FL 


16 


Eastern Carolina SDA, NC 


16 


Polk Workforce Development 


16 


Cincinnati City in Hamilton 


16 


Workforce Council of South 


16 


Montgomery/Preble/Dayton 


16 


Honolulu City/County, HI 


16 


Portage County, OH 


16 


Mayor's Office of E & T, IL 


16 


Ohio 18 Consortium, OH 


16 


Northeastern Indiana SDA, IN 


16 


Northern Tier Consortium, PA 


16 


Rapides Parish, LA 


9 


SDA # 9 Catawba SDA, SC 


16 


Louisiana District I 


16 


JSTD 5. Cleveland, TN 


16 


Baltimore County, MD 


16 


City of Richmond, VA 


13 


Prince Georges County, MD 


16 


The Pacific Mountain, WA 


16 


Northern Middlesex Consortium, 


16 


Northeast Consortium, PR 


16 
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APPENDIX III 

U.S. Department of Labor Employment and Training Administration 

200 Constitution Avenue, N.W. 
Washington, D.C. 20210 

MAR 7 2001 



MEMORANDUM FOR JOHN J. GETEK 

Assistant Inspector General for Audit 



FROM: RAYMOND J. UHALDE hyetfdJfr- 

Deputy Assistant Secretary for Employment and Training 

SUBJECT: Performance Audit of Job Training Partnership Act, Title 

II-C, Out-of-School Youth Program Draft Audit Report 
No. 06-01-001-03-340 




Thank you for providing the subject report and the opportunity to provide comments. I 
want to thank you and your audit staff for your willingness to work with our program 
staff in completing this draft audit report. Overall, we found the report to contain 
findings that were clearly articulated, well-presented, and supported by the facts derived 
from the sample you reviewed. 

We agree that performance results must be credible. If the reality or the perception is 
that programs are not reporting accurately, the system's integrity is at stake. Therefore, 
the Employment and Training Administration (ETA) is committed to developing clear, 
precise, and feasible standards for documenting program outcomes. ETA's new data 
validation initiative is attempting to create both more precise programming 
specifications and also more precise standards for validating data quality. ETA expects 
the validation system to be designed and pilot-tested by September, 2002. 

Following is our response to each of the specific recommendations in the report as well 
as additional comments about the findings contained in this report. 

RECOMMENDATIONS: 

• Notify all States and substate WIA grantees of the results of this audit and 

emphasize the necessity for states and their grantees to include outcomes data 
validation as part of their monitoring programs and service provider certification 
programs. 
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• Emphasize to the states and substate grantees the importance of documenting not 
only reported outcomes but also specific services provided, dates services were 
provided, and actual program exit date - date of last activity. 

RESPONSE: 

• When the report is final, ETA will issue a program guidance letter informing the 
States of the findings and recommendations contained in this report. States will 
be asked to inform their WIA substate grantees of the audit findings. In addition, 
we routinely post our formal guidance letters on our web site. In this same 
directive, we will encourage States and their grantees to include outcome data 
validation as part of their program reviews. Note that while local Boards are 
responsible for oversight of providers, there is no requirement for "service 
provider certification" under WIA. We will recommend that subcontracts with 
service providers include a provision for maintaining accurate case records, 
including recording of outcome data. 

• The audit report cites several legislative and other federal guidelines that mention 
documentation requirements to support. However, the report does not clearly 
identify the standards of documentation that were applied. It would be helpful if 
the report could do so and could acknowledge the extent to which these standards 
are consistent with previous guidance issued under either JTPA or WIA. 



RECOMMENDATION: 

• Include in ETA's monitoring program a review of the states' and substate WIA 
grantees' implementing such a data validation effort. 

RESPONSE: 

• Section 667.400 of the WIA regulations outline federal, state and local 
responsibilities for oversight and monitoring. Section 667.400 (a) stipulates that 
Federal oversight will be conducted primarily at the recipient (i.e. State) level. 
Section 667.400(c) outlines recipient and subrecipient monitoring of 
grant-supported activities. Under WIA regulations, monitoring of subrecipient or 
substate activities is primarily a State responsibility. 

• At the federal level, ETA's regional staff have primary responsibility for ongoing 
program oversight and monitoring. We will convey the findings of the report to 
our regional staff and instruct them to work with their States in implementing 
data validation efforts as part of the States' oversight of their WIA substate 
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grantees. We will also instruct regional staff to include this area in any program 
reviews at the State level. 

RECOMMENDATION: 

• Emphasize the importance of the WIA program's not only enrolling youth in 
occupational skills training activities, but also finding ways to keep them actively 
participating in the program to completion, to maximize their post-program 
earnings. 

RESPONSE: 

• We will include this emphasis in the above mentioned program guidance letter 
disseminating the findings of the audit. We also issue program guidance 
information each year, typically about the summer employment opportunities 
program. This year we plan to issue comprehensive guidance on youth services 
under WIA and will include a section in this guidance letter on retention of 
participants. While we concur with the need to focus on retention strategies, it is 
also important to understand that these programs serve some of the most 
challenging youth. Many economically disadvantaged, out-of- school youth face 
multiple barriers to successful transition into the work force. In addition, for 
many youth, the process of moving from adolescence to adulthood is characterized 
by numerous impediments and it is only over time that they mature and become 
productive citizens. 

ADDITIONAL COMMENTS: 

The report states that 38 terminees were found to have no wage records and the 
"employer confirmed (they) did not employ (the) participant." There is no evidence 
provided in the audit report of the thoroughness of attempts to obtain verification of two 
year old employment data from the employer. Given that there may be multiple employer 
sites, turnover in employment staff, and other factors that could result in a false negative 
outcome, it is difficult to determine the accuracy of this specific audit finding. The report 
partially acknowledges this by indicating that "some of the outcomes that were 
documented may have, in fact, been attained." We recommend that the report explain the 
steps taken to verify employment outcomes and acknowledge the difficulty of verifying 
employment outcomes not reflected in the wage record system some two years after they 
are reported. 

We also suggest that the report be revised to refer to the JTPA regulatory requirements 
in the past tense. It is confusing as currently drafted. Some examples are on pages 4, 5, 
and 9. 



Office of Inspector General 



JTPA’s Title Il-C Out-of-School Youth Performance Audit 



Unfortunately the solution to documentation and verification issues is not as simple as 
issuing more explicit guidelines or conducting additional data validation reviews. We 
are mindful that a core principle of the reforms under WIA is to improve services to 
youth. While we need to insure the integrity of our reported program outcomes, we also 
need to acknowledge the administrative burden imposed on the programs to collect 
extensive documentation. An additional problem is the burden on the employer 
community. Employers may be reluctant to work closely with workforce development 
programs if they perceive too great a burden in providing evidence of employment. It 
could be counterproductive to the purposes of WIA if administrative requirements 
become so burdensome that they are a deterrent to the delivery of program services. 

Again, we appreciate the opportunity to work with you in preparing this report as well as 
the opportunity to provide these comments. Please contact me if you need further 
information. 
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